Skip to content

Preserve requires_grad in SimpleFSDP parameter registration - #4282

Open
sylvesterkaczmarek wants to merge 1 commit into
pytorch:mainfrom
sylvesterkaczmarek:fix/simple-fsdp-preserve-requires-grad
Open

Preserve requires_grad in SimpleFSDP parameter registration#4282
sylvesterkaczmarek wants to merge 1 commit into
pytorch:mainfrom
sylvesterkaczmarek:fix/simple-fsdp-preserve-requires-grad

Conversation

@sylvesterkaczmarek

Copy link
Copy Markdown
Contributor

Fixes #3885.

Summary

SimpleFSDP re-registers each distributed parameter by wrapping it in nn.Parameter(...). Because nn.Parameter defaults requires_grad=True, a parameter that was intentionally frozen before SimpleFSDP wrapping becomes trainable again.

This change passes the original parameter's requires_grad value when re-registering the distributed parameter, preserving both frozen and trainable parameter state.

Changes

  • preserve p.requires_grad when constructing the replacement nn.Parameter in data_parallel()
  • extend the existing single-rank SimpleFSDP test to cover a model with a frozen weight and trainable bias
  • verify both the underlying registered parameters and parametrized attribute access preserve their original requires_grad values

The behavior of already-trainable parameters is unchanged.

Validation

Added test_preserves_parameter_requires_grad to torchtitan/experiments/graph_trainer/tests/test_simple_fsdp.py. It exercises the existing public SimpleFSDP application path at world size 1 with the CPU/gloo test setup already used by this file.

The test suite and pre-commit were not run locally in this environment because the repository is not available as a local checkout here. No local test result is claimed; CI should run the updated test file.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Aug 22, 2026
@tianyu-l

Copy link
Copy Markdown
Contributor

not sure if the graph trainer error is introduced by this change

@sylvesterkaczmarek

Copy link
Copy Markdown
Contributor Author

not sure if the graph trainer error is introduced by this change

Thanks. I checked the current run. Lint passed, while the GraphTrainer integration job is the only failing signal I found. I don't have Actions write permission on the upstream repo to rerun it.

Could you rerun the failed job so we can confirm whether this is transient before changing the approved one-line fix?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add the required_grads of parameter to register_parameter

2 participants